Now, moving ahead from comparing the average values between two or more groups,
we are looking at two variables.
We want to know if there is a statistically significant correlation
between these two variables and what is needed for this to happen.
We would need to look back to the earlier definition of types of variables.
We generally define the variables in two groups, the categorical variables,
an continuous variables.
So if we were to go back to our teaching ratings data,
we have instructors who are male and female.
Some instructors are visible minorities and some are Caucasian.
So we have two variables, male, and female and the visible minority status.
These two variables are examples of categorical variables, and
if we're comparing or trying to determine the correlation between two categorical
variables, we would use the Chi-square Ttest.
We would begin with a cross tabulation between the two values. If we have two
continuous variables, for example, the teaching evaluations score and
the beauty score of an instructor, then these are two continuous variables and
they can assume any reasonable value within the range.
In this case, we use a Pearson correlation test.
We usually begin with a scatter plot to see what's the nature of the relationship
between the two variables.
Let us start with categorical variables.
We will use the Chi-square Test for Association.
First, we state our hypothesis.
We will test the null hypothesis that gender an tenureship are independent
against the alternative hypothesis that they're associated.
Let's begin with a cross tabulation between gender male and female, and
tenure.
That is, tenured profs then followed by a Chi-square test.
So we do the tabulations.
In the rows we have tenured no versus tenured yes, and
female instructors and males.
We would like to eyeball these numbers before we turn them into percentages.
Looking at instructors who are non-tenured,
we notice that 50 of the instructors are female versus 52 who are male.
But for the instructors who are tenured, 145 of them are female,
and 216 of them are male.
So within the tenured group we see greater probability for males to be tenured, but
in the untenured group the distribution between males and females look similar.
Before we go to Python, let's do this by hand to understand the concept.
The formula for Chi-square is given as follows.
The summation of the observed value, i.e the counts in each group minus
the expected value, all squared, divided by the expected value.
Expected values are based on the given totals.
What would we say each individual value would be if we did not know the observed
values?
So to calculate the expected value of untenured female instructors,
we take the row total, which is 102 multiplied by the column total 195,
divided by the grand total of 463.
This will give you 42.96.
If we do the same thing for tenured male instructors,
we will take the row total 361 multiplied by the column
total 268 divided 463, we get 208.96..
If we repeat the same procedure for all of them, we get these values.
If we take the row totals, column totals, and grand total,
we will get the same values as the totals as the observed values.
Now going back to this formula,
if we take a summation of all the observed minus the expected values, all squared,
divided by the expected value, we will get a Chi-square value of 2.557.
And the degree of freedom will be 1.
On the Chi-square table, we check the degree of freedom equals row one and
find the value closest to 2.557.
Here we can see that 2.557 will most likely
fall in between a p-value of 0.1 and 0.25.
Therefore, we can say that the p-value is greater than 0.1.
Since the p-value is greater than 0.05,
we fail to reject the null hypothesis that the two variables are independent and
therefore we will conclude that the alternative hypothesis that there is
an association between gender and tenureship does not exist.
To do this in Python we will use the Chi-square contingency function in
the SciPy statistics package, that is a Chi-square test value of 2.557.
And the second value is the p-value of about 0.11.
And a degree of freedom of 1.
If you remember, the Chi-square table did not give an exact p-value but
a range in which it falls.
Python will give the exact p-value.
We can see the same results as on the previous slides.
It also prints out the expected values, which we also calculated by hand.
Since the p-value is 0.11, which is greater than 0.05,
we fail to reject the null hypothesis that the two variables are independent.
And therefore we will conclude the alternative hypothesis
that there is an association between gender and tenureship does not exist.
This was an example of testing independence between two categorical
variables.
Now to continuous variables using a Pearson correlation test from the teaching
ratings data.
We will test the null hypothesis that there is no correlation between
an instructor's beauty score and their teaching evaluation score against
the alternative hypothesis that there is a correlation between both variables.
We had the normalized beauty score on the x-axis and
the teaching evaluation score on the y-axis.
You can eyeball a positive upward sloping curve, but
let's run a Pearson correlation test to find out.
We will use the Pearson R package in the scipy.stats package and check for
the correlation.
We will get a coefficient value of how strong the relationship is and
in what direction.
Correlation coefficient values lie between -1 and 1.
Where -1 means a strong negative correlation and
visually represented by a downward sloping curve, and 1 means a strong
positive relationship and visually represented by an upward sloping curve.
In our case, we have a Pearson coefficient of 0.18,
and a p-value of 4.25 times 10 raised to power -5.
Since the p-value is less than 0.05, we reject the null hypothesis and
conclude that there exists the relationship between an instructor's
beauty score and teaching evaluation score.
[MUSIC]